Pitfalls in research on ecological validity of novel executive function tests: A systematic review and a call to action.

The term "ecological validity" (EV) has traditionally referred to test scores' ability to predict real-world functioning. However, a test's similarity to real-world tasks is sometimes mistaken for evidence of its ability to predict daily life, sometimes bypassing rigorous validation research. The goal of this systematic review was to examine the type and quality of evidence used to support claims of EV of novel face-valid tests of executive functions (EF). MEDLINE and PsychINFO databases were searched using the following terms: ecologic* AND neuropsychol* AND (executive function* OR executive dysfunction OR executive abilit*). Thirty-two articles that explicitly stated that their results demonstrated EV of a novel face-valid test of EF were identified. Results showed that only 60% of studies based their claims about EV on test scores' ability to predict functional outcomes, with the remaining 40% relying on other evidence (e.g., correlations with other measures, participant feedback, group differences). Among the studies that did base their conclusions on test scores' ability to predict outcomes (n = 19), an overwhelming majority relied on behavioral rating scales, utilized small sample sizes and participant-to-variable ratios, and failed to control for covariates and multiple comparisons. Poor scientific rigor was particularly pronounced in studies of "naturalistic" tests. The present systematic review reveals significant conceptual, methodological, and statistical flaws among an overwhelming majority of studies that claim to have found support for the EV of a novel face-valid test of EF. We call upon authors, reviewers, and editors to safeguard the scientific rigor of research in this area. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Test of Practical Judgment (TOP-J): Construct, Criterion, and Incremental Validity in a Community Sample of Older Adults.

OBJECTIVE: The Test of Practical Judgment (TOP-J) is a stand-alone judgment measure that is considered to tap into aspects of executive functioning (EF) and inform clinical predictions of daily functioning in older adults. Past validation research is variable and has some limitations. The present study sought to examine the reliability and construct, criterion, and incremental validities of scores on TOP-J 9-item version (TOP-J/9). METHOD: Participants were 95 community-dwelling older adults aged 60 to 85. Participants completed TOP-J/9, measures of EF and global cognition, and three different modalities of instrumental activities of daily living (IADLs) (self-report, performance-based tasks conducted in the laboratory, and performance-based tasks completed at home over 3 weeks). RESULTS: TOP-J/9 scores showed adequate internal consistency (α = 0.73) after correcting for the low number of items. TOP-J/9 was correlated with global cognition and EF, although EF did not survive correction for lower-order processes. Finally, although TOP-J/9 scores were associated with home-based IADL tasks (but not with self-report and laboratory-based IADLs), providing some evidence of criterion validity, they did not incrementally contribute to home-based IADL performance beyond other cognitive measures. However, when two items pertaining to social/ethical judgment were removed, this modified version of TOP-J did relate to EF beyond lower-order processes and contributed uniquely to prediction of home-based IADLs beyond other measures. CONCLUSION: Results suggest that TOP-J/9 taps into global cognitive status (but not necessarily EF) and predicts "real-world" functioning (but not above and beyond other cognitive measures). TOP-J psychometrics may be improved by removing two social/ethical items.

Know thyself: Executive functioning and sex predict self-appraisal of functional abilities in community-dwelling older adults.

Objective: The capacity for accurate self-appraisals of age-related changes in cognitive and functional abilities is integral to the maintenance of independence in later life, yet there is little understanding of the factors that place nondemented older adults at risk for poor self-awareness. This study examined the potential contributions of executive functioning (EF), crystallized intelligence (IQ-Cr), and sex in predicting congruence between performance and self-appraisals of instrumental activities of daily living (IADL) in a group of community-dwelling older adults. Method: A group of 150 nondemented, community-dwelling older adults (White and majority highly educated) completed measures of EF and IQ-Cr. Participants also completed five timed IADL tasks and self-appraised their performance relative to others of similar age. Results: Sex [F(1,148) = 7.75, p = .006, ηp2 = .05] and EF [(F(1,147) = 5.30, p = .02, ηp2 = .04)], but not IQ-Cr, predicted the relationship between performance and self-appraisals, such that those with lower EF and those of male sex overestimated their performance more than those with higher EF and females. Conclusions: Findings indicate that having average to below average EF abilities and being of male sex are risk factors for less accurate self-report of IADL abilities and as such might represent important considerations when assessing IADL abilities via self-report among largely independent, community-dwelling older adults.